Music charts around the world give insights into the ever changing music tastes of the population which they represent. In Australia, the Australian Recording Industry Association (ARIA) have been publishing weekly rankings of tracks and albums selling in the Australian market under the name of the ARIA Charts since 1970. Triple J, a national Australian radio station intended to appeal to listeners of alternative music, also host an annual ranking event titled the Triple J Hottest 100, a vote-driven chart of the 100 most popular songs amongst its listeners.
Our team aims to investigate how Australian music tastes have changed over time by comparing the audio features of tracks from the yearly ARIA and Triple-J Hottest 100 charts from 1970-2019. We will be using the Spotify API to map the tracks from the charts to its corresponding Spotify URI, allowing us to analyse individual tracks against key attributes: duration, key, mode, time signature, acousticness, speechiness, danceability, energy, loudness, valence/happiness and tempo.
The documentation for the endpoint we are using to get the analysis data is available here. We are sourcing the ARIA charts and Triple J Hottest 100 rankings from these sites:
This dashboard uses Plotly to allow you to interact with our graphs. Use the tools when hovering over charts to select and filter attributes or focus in on time ranges!
Our Discove-R-Weekly team as part of ETC1010:
The data was sourced through calling the above endpoint of the Spotify API. This was done by using their Javascript client libraries since it was what we were most familiar with.
The source code for pulling data from Spotify is available on Github here.
We created or sourced a Spotify playlists which included the songs for each year’s charts from 1970 - 2019. This playlist could then be passed to the CLI tool.
Given a list of playlist URIs or URLs in a .txt file, the program will get the tracks associated with that playlist, and call the Spotify Audio Features endpoint to get insightful information about the tracks.
This data was outputted in csv format into the following structure:
└───data
│ playlists.csv
│
├───features
│ 0hIiy3ihpzsIX9Dd6RVtWw.csv
│ 0kgHtoYJSMS3pMMciC3Us4.csv
│ ...
│
├───artists
│ 0hIiy3ihpzsIX9Dd6RVtWw.csv
│ 0kgHtoYJSMS3pMMciC3Us4.csv
│ ...
│
└───tracks
0hIiy3ihpzsIX9Dd6RVtWw.csv
0kgHtoYJSMS3pMMciC3Us4.csv
...
playlists.csv : a lookup of the playlist URI and the playlist name and description/features: folder with an individual csv for each playlist, containing the audio features data./artists: folder with an individual csv for each playlist, containing data about each artist (can be multiple for each track)./tracks: folder with an individual csv for each playlist, containing data about each track.playlists.csvHas consolidated information about each playlist.
Primary Key:
playlist_uri: Spotify URI for the playlist. Also the title of corresponding CSV files.Other Attributes:
title: Title of the playlistdescription: Description of the playlisturl: URL of the playlistAfter cleaning & consolidation:
year: Year of the chartchart: H100, ARIA or OTHERfeatures/...Has audio features for each track returned from the Spotify Audio Features endpoint.
Primary Keys: - playlist_uri: Spotify URI for the playlist - uri: Spotify URI of the track
Other Attributes: The following features are saved in the CSV. Read the description and distribution for each attribute from the above Spotify API documentation.
duration_mskeymodetime_signatureacousticnessdanceabilityenergyinstrumentalnesslivenessloudnessspeechinessvalencetempotracks/...Idenitifying information about each track, as returned from the Spotify Get Tracks endpoint.
Note: Tracks can have multiple artists. The CSV has been formatted to have an entry for each listed artist of the track.
Primary Keys:
playlist_uri: Spotify URI for the playlisturi: Spotify URI of the trackartist: name of a featuring artistartist_uri: Spotify URI of the artistOther Attributes:
The following features are saved in the CSV. Read the description and distribution for each attribute from the above Spotify API documentation.
album: name of the albumalbum_uri: Spotify URI for the albumdisc_numberduration_msnamepopularityexpliciturilink: renamed from hrefartists/...Idenitifying information about each artist, as returned from the Spotify Get Artists endpoint.
Note: Genres for each artist are comma separated and should be expanded
Primary Keys:
uri: Spotify URI of the artistOther Attributes:
The following features are saved in the CSV. Read the description and distribution for each attribute from the above Spotify API documentation.
name: name of the artistfollowers: Number of followerspopularityurigenres: comma separated string of genreslink: renamed from hrefAfter the above process, we had to get the data into R and create our dataframes. Since the data coming from the Spotify API is reliable and clean, most of the work involves merging together the data from the separate csvs into a single dataframe. Our process is outlined below.
First, let’s get the playlists.csv file which has information about all the playlists included in the dataset.
playlist_data <- read_csv(here::here('Data/playlists.csv'))
features_data <-
list.files(path = here::here('Data/features'),
pattern = "*.csv",
full.names = T) %>%
map_df(~read_csv(.))
tracks_data <-
list.files(path = here::here('Data/tracks'),
pattern = "*.csv",
full.names = T) %>%
map_df(~read_csv(.))
artists_data <-
list.files(path = here::here('Data/artists'),
pattern = "*.csv",
full.names = T) %>%
map_df(~read_csv(.))
glimpse(playlist_data)Rows: 77
Columns: 4
$ playlist_uri <chr> "spotify:playlist:6YKI2VYSO9iZtyaLb3ZUsG", "spotify:play…
$ title <chr> "ARIA Top 100 Singles of 1970", "ARIA Top 100 Singles of…
$ description <chr> "Unavailable: Lionel Rose - I Thank You, Mary Hopkin - K…
$ url <chr> "https://api.spotify.com/v1/playlists/6YKI2VYSO9iZtyaLb3…
Rows: 7,605
Columns: 15
$ playlist_uri <chr> "spotify:playlist:03vMytYzGsKy1dQkGDdiCp", "spotify:…
$ duration_ms <dbl> 269667, 180566, 229526, 241693, 295502, 176561, 2535…
$ key <dbl> 0, 4, 10, 4, 5, 7, 5, 1, 5, 2, 1, 9, 1, 9, 2, 0, 4, …
$ mode <dbl> 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0…
$ time_signature <dbl> 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4…
$ acousticness <dbl> 0.00801, 0.16600, 0.36900, 0.63400, 0.32900, 0.00346…
$ danceability <dbl> 0.856, 0.782, 0.689, 0.566, 0.470, 0.723, 0.489, 0.5…
$ energy <dbl> 0.609, 0.685, 0.481, 0.664, 0.431, 0.809, 0.597, 0.8…
$ instrumentalness <dbl> 8.15e-05, 1.18e-05, 1.03e-06, 0.00e+00, 0.00e+00, 1.…
$ liveness <dbl> 0.0344, 0.1600, 0.0649, 0.1160, 0.0854, 0.5650, 0.10…
$ loudness <dbl> -7.223, -6.237, -7.503, -5.303, -6.129, -3.081, -6.6…
$ speechiness <dbl> 0.0824, 0.0309, 0.0815, 0.0464, 0.0342, 0.0625, 0.02…
$ valence <dbl> 0.928, 0.603, 0.283, 0.437, 0.289, 0.274, 0.324, 0.6…
$ tempo <dbl> 114.988, 118.016, 80.025, 128.945, 157.980, 98.007, …
$ uri <chr> "spotify:track:32OlwWuMpZ6b0aN2RZOeMS", "spotify:tra…
Rows: 8,893
Columns: 12
$ playlist_uri <chr> "spotify:playlist:03vMytYzGsKy1dQkGDdiCp", "spotify:play…
$ album <chr> "Uptown Special", "Uptown Special", "Me 4 U", "Me 4 U", …
$ album_uri <chr> "spotify:album:3vLaOYCNCzngDf8QdBg2V1", "spotify:album:3…
$ artist <chr> "Mark Ronson", "Bruno Mars", "OMI", "Felix Jaehn", "Wiz …
$ artist_uri <chr> "spotify:artist:3hv9jJF3adDNsBSIQDqcjp", "spotify:artist…
$ disc_number <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,…
$ duration_ms <dbl> 269666, 269666, 180565, 180565, 229525, 229525, 241693, …
$ name <chr> "Uptown Funk (feat. Bruno Mars)", "Uptown Funk (feat. Br…
$ popularity <dbl> 81, 81, 76, 76, 82, 82, 74, 73, 60, 60, 60, 71, 80, 77, …
$ explicit <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, …
$ uri <chr> "spotify:track:32OlwWuMpZ6b0aN2RZOeMS", "spotify:track:3…
$ link <chr> "https://api.spotify.com/v1/tracks/32OlwWuMpZ6b0aN2RZOeM…
Rows: 8,834
Columns: 6
$ name <chr> "Mark Ronson", "Bruno Mars", "OMI", "Felix Jaehn", "Wiz Kh…
$ followers <dbl> 849349, 27072330, 552534, 1040438, 9412211, 11554345, 4205…
$ popularity <dbl> 79, 89, 69, 82, 88, 84, 82, 85, 83, 75, 83, 84, 74, 95, 91…
$ uri <chr> "spotify:artist:3hv9jJF3adDNsBSIQDqcjp", "spotify:artist:0…
$ link <chr> "https://api.spotify.com/v1/artists/3hv9jJF3adDNsBSIQDqcjp…
$ genres <chr> "dance pop,pop", "dance pop,pop", "dance pop", "dance pop,…
We need to extract the year of the playlist and also whether it was for the Hottest 100 of ARIA Chart. We can give these the flags H100 and ARIA.
playlist_data <- playlist_data %>%
mutate(year = as.numeric(str_extract(title, "\\d{4}$"))) %>%
mutate(chart = as.factor(if_else(
grepl("ARIA", title, fixed = TRUE),
"ARIA",
if_else(grepl("Hottest 100", title, fixed = TRUE),
"H100",
"OTHER")
)))
head(playlist_data) %>% tibble()# A tibble: 6 x 6
playlist_uri title description url year chart
<chr> <chr> <chr> <chr> <dbl> <fct>
1 spotify:playlist… ARIA Top … Unavailable: Lionel R… https://api.s… 1970 ARIA
2 spotify:playlist… ARIA Top … Unavailable: Lally St… https://api.s… 1971 ARIA
3 spotify:playlist… ARIA Top … Unavailable: Slim New… https://api.s… 1972 ARIA
4 spotify:playlist… ARIA Top … Unavailable: Jamie Re… https://api.s… 1973 ARIA
5 spotify:playlist… ARIA Top … Unavailable: Sister J… https://api.s… 1974 ARIA
6 spotify:playlist… ARIA Top … Unavailable: Bob Huds… https://api.s… 1975 ARIA
We can now write the consolidated dataframes to csvs for submission.
Let’s first have a look at what the charts are like in the present day. The 2020 charts have not been finalised yet, so we’re analysing the top 100 songs from the ARIA Chart of 2019.
The table depicts the Top 10 artists from year 2019 that appeared on the Top 100s ARIA chart. From the table we can conclude that Post Malone is the most popular artist with 7 tracks released on 2019. Following Post Melone, Ed Sheeran and Khalid are the next top artists with 6 tracks published in the year 2019. Conversely, artists with 3 tracks published on the year 2019 made to the bottom list of top 10.
Evidently, the most prevalent genre in 2019 was pop, doubling the presence of the next most seen genre. A significant proportion of pop in 2019 can be potentially attributed to album releases of popular pop-centric artists such as Taylor Swift, Ed Sheeran, Billie Eilish and Ariana Grande. Each of the aforementioned artists had at least three tracks in the Top 100 Chart, totalling 17 tracks, making up 17% of the Top 100 Chart. Besides pop and its derivatives such as dance pop and post-teen pop, rap and EDM are present in the top 10 with artists such as Post Malone bolstering the Rap genre; and Marshmello and Calvin Harris helping to keep EDM within the Top 10.
We can see from this figure that for songs that were popular in 2019, the median levels of danceability and energy were relatively high, compared to levels of acousticness, instrumentalness, liveness and speechiness. This indicates that Australians generally preferred songs that feel energetic and are suitable for dancing. When looking at the median level of valence and the whiskers, there seems to be an approximately even split between preference for songs that sound more positive compared to songs that sound more negative. There is greater variance among the middle 50% of the data for valence in particular, and this is slightly skewed towards the lower values, indicating the existience of a slightly higher concentration of popular songs in 2019 that sounded more negative.
Since the 1970s, technology, culture and fashion have changed drastically. Australian music trends have also changed as popular music trends in the 1970s are rarely seen in the present day. We have investigated genres, artists and audio features (duration, key, mode, time signature, acousticness, speechiness, danceability, energy, loudness, valence/happiness and tempo) from 1970 to 2019 to analyse these changes. Let’s take a look at how the ARIA charts have evolved over time. Here’s what the ARIA chart looked like in 1970:
The animation below shows the rise and fall of genres on the ARIA chart between 1970 and present. At the beginning, genres such as soft rock and folk rock dominated the chart. The 80’s were all about new wave pop with europop make an entrance in the 90s.
With the turn of the millenium came the dominance of pop in the charts in all its variations. We see new genres like hip-hop and rap also make an entrance. EDM begins to enter the chart around the turn of the 2010s, and with it came electro-pop and a bunch electronic variations such as house.